Skip to content

In-house discv5#11728

Merged
flcl42 merged 214 commits into
masterfrom
generic-kad-2
Jun 23, 2026
Merged

In-house discv5#11728
flcl42 merged 214 commits into
masterfrom
generic-kad-2

Conversation

@flcl42

@flcl42 flcl42 commented May 22, 2026

Copy link
Copy Markdown
Contributor

Changes

  • Even more generic KAD
  • Our own discv5 that shares code with discv4
  • Optimized and checked for vulnerabilities

Types of changes

What types of changes does your code introduce?

  • Bugfix (a non-breaking change that fixes an issue)
  • New feature (a non-breaking change that adds functionality)
  • Breaking change (a change that causes existing functionality not to work as expected)
  • Optimization
  • Refactoring
  • Documentation update
  • Build-related changes
  • Other: Description

Testing

Requires testing

  • Yes
  • No

If yes, did you write tests?

  • Yes
  • No

Remarks

Change list

  • Extracted the reusable Kademlia routing logic into Nethermind.Kademlia: routing table, lookup algorithm, node health tracking, distance abstractions, and generic node/key operators now live outside discv4-specific code.
  • Rewired discv4 to consume the shared Kademlia core through a protocol adapter, while keeping discv4-specific bonding, signed RLP messages, FINDNODE/NEIGHBORS, ENR request/response, and XDC extension hooks in the discv4 layer.
  • Replaced the external Lantern-based discv5 implementation with native Nethermind code: UDP packet codec, WHOAREYOU handshake, session/key derivation, message codec, PING/PONG, FINDNODE/NODES, TALKREQ/TALKRESP, and ENR propagation.
  • Added a separate discv5 Kademlia instance and adapter so discv5 FINDNODE distance requests map to GetAllAtDistance, while discv4 keeps nearest-node lookup semantics.
  • Unified discovery lifecycle shape: discv4 and discv5 both run through discovery app lifecycle, child Autofac protocol scopes, Kademlia node sources, persistence, channel activation, and node-removal handling.
  • Kept execution peer candidates separate from discovery routing endpoints: discv5 routing uses ENR UDP endpoints, while peer-pool candidates are emitted only from usable ENR TCP endpoints.
  • Hardened ENR handling: preserved unknown ENR entries, rejected consensus-only eth2 records before execution peer admission, skipped malformed records, and prevented stale ENR sequence downgrades.
  • Hardened UDP receive paths and hot allocations: cheap pre-copy packet checks, global/per-IP discv4 inbound limiting, bounded receive queues, pooled/owned discv5 message decode, direct RLP length calculations, and reduced serializer buffer copies.
  • Bounded discv5 state that can grow under remote input: sessions, sent challenges, pending nonces, response handlers, known ENRs, recent-node dedupe, and NODES response accumulation.
  • Improved shutdown/disposal behavior so discovery node-source enumeration stops background discovery work when the caller disposes enumeration without externally cancelling.
  • Refactored discovery code organization for reviewability: discv4-specific messages/serializers/handlers stay in discv4 folders, discv5 serializers are per-message, concrete leaf types are sealed where not used as extension hooks.
  • Updated network defaults and smoke-validated behavior: mainnet uses both discoveries, testnets use V5-only, and mainnet/Hoodi runs showed discv5 discovery traffic without discv5 packet-loop errors.
  • Added focused interoperability and regression coverage, including go-ethereum discv5 wire vectors, packet codec/session tests, serializer round-trips, Kademlia adapter/source tests, ENR endpoint filtering tests, and discovery lifecycle/disposal tests.

Architecture overview

The new shape separates reusable Kademlia table mechanics from protocol transport. Nethermind.Kademlia owns generic routing, lookup, health tracking, and distance services; Nethermind.Network.Discovery supplies Ethereum discovery-specific adapters, persistence, node sources, and lifecycle. Discv4 and discv5 each create their own protocol child scope and own Kademlia instance, then bind protocol-specific message senders and node-source behavior on top of the same core. Discv4 keeps signed RLP packet handling, bonding, and nearest-neighbour discovery, while discv5 owns packet/session encryption, ENR-based messages, and distance-bucket FINDNODE semantics. Both versions feed discovered execution peer candidates through role-specific ENR conversion so UDP discovery endpoints stay in routing and TCP endpoints go to the peer pool.

High-level architecture

flowchart TB
    NetworkConfig["Network config<br/>DiscoveryVersion + bootnodes"] --> Composite["CompositeDiscoveryApp"]
    Composite --> V4App["Discv4 DiscoveryApp"]
    Composite --> V5App["Discv5 DiscoveryV5App"]

    V4App --> V4Scope["discv4 child scope"]
    V5App --> V5Scope["discv5 child scope"]

    V4Scope --> V4Adapter["discv4 KademliaAdapter<br/>bonding + FINDNODE nearest lookup"]
    V5Scope --> V5Adapter["discv5 KademliaAdapter<br/>session-aware FINDNODE distances"]

    V4Adapter --> Kad4["Kademlia<PublicKey, Node><br/>discv4 instance"]
    V5Adapter --> Kad5["Kademlia<PublicKey, Node><br/>discv5 instance"]

    Kad4 --> Core["Nethermind.Kademlia<br/>routing table + lookup + health"]
    Kad5 --> Core

    V4App --> V4Handler["NettyDiscoveryHandler<br/>signed RLP UDP packets"]
    V5App --> V5Handler["NettyDiscoveryV5Handler"]
    V5Handler --> PacketCodec["PacketCodec<br/>WHOAREYOU + sessions + AES-GCM"]
    PacketCodec --> MessageCodec["MessageCodec<br/>PING/PONG/FINDNODE/NODES/TALK"]

    V4Adapter --> NodeSource4["discv4 NodeSource"]
    V5Adapter --> NodeSource5["discv5 NodeSource"]
    NodeSource4 --> PeerPool["PeerPool candidates"]
    NodeSource5 --> PeerPool
Loading

Key moment: discv5 packet handling

sequenceDiagram
    participant UDP as NettyDiscoveryV5Handler
    participant Codec as PacketCodec
    participant Adapter as Discv5 KademliaAdapter
    participant Kad as Kademlia

    UDP->>UDP: cheap size checks and bounded queue admission
    UDP->>Codec: accepted packet bytes
    Codec->>Codec: decode ordinary, WHOAREYOU, or handshake packet
    alt no usable session
        Codec-->>UDP: send WHOAREYOU challenge
    else authenticated message
        Codec->>Adapter: Discv5Message
        Adapter->>Kad: mark inbound health / add or refresh node
        alt FINDNODE
            Adapter->>Kad: GetAllAtDistance(requested distances)
            Kad-->>Adapter: matching nodes
            Adapter-->>Codec: NODES responses with ENRs
        else PING/PONG/TALK
            Adapter-->>Codec: protocol response or handler completion
        end
        Codec-->>UDP: encrypted response packet
    end
Loading

Key moment: ENR endpoint separation

flowchart LR
    ENR["ENR"] --> Validate["validate secp256k1<br/>node id, routability, sequence"]
    Validate --> Eth2{"consensus-only<br/>eth2 record?"}
    Eth2 -- yes --> Reject["reject for execution discovery"]
    Eth2 -- no --> Record["NodeRecord"]

    Record --> Routing["TryFromDiscoveryEnr<br/>UDP endpoint"]
    Routing --> KadTable["discv5 Kademlia table"]

    Record --> Candidate{"has usable TCP endpoint?"}
    Candidate -- yes --> PeerNode["TryFromEnr<br/>TCP endpoint"]
    PeerNode --> PeerPool["PeerPool"]
    Candidate -- no --> SkipPeer["skip RLPx peer candidate"]

    Record --> Sequence{"newer ENR sequence?"}
    Sequence -- yes --> Cache["replace known ENR"]
    Sequence -- no --> Keep["keep newer cached ENR"]
Loading

Key moment: receive-path hardening

flowchart TD
    Packet["UDP packet"] --> Cheap["read size/type from buffer"]
    Cheap --> Valid{"valid enough<br/>for this protocol?"}
    Valid -- no --> DropOrForward["drop or forward without copy"]
    Valid -- yes --> Limit["global + per-IP limits"]
    Limit --> Accepted{"within limits<br/>and queue capacity?"}
    Accepted -- no --> Drop["drop before packet copy/expensive work"]
    Accepted -- yes --> Queue["bounded channel"]
    Queue --> Workers["limited dispatch workers"]
    Workers --> Decode["deserialize / decrypt accepted traffic"]
    Decode --> Adapter["protocol adapter"]
    Adapter --> Kad["Kademlia + response handlers"]
Loading

flcl42 and others added 6 commits June 15, 2026 12:08
# Conflicts:
#	src/Nethermind/Nethermind.Runner/configs/hoodi_archive.json
#	src/Nethermind/Nethermind.Runner/configs/mainnet_archive.json
#	src/Nethermind/Nethermind.Runner/configs/sepolia_archive.json
…SZ ulong limits, injectable discv5 ENR filter) (#11985)

* feat(ssz): support list limits beyond int range in the SSZ generator

Beacon-chain lists use limits up to 2^40 (VALIDATOR_REGISTRY_LIMIT), which
overflowed the int-typed SszListAttribute. The limit is now ulong end to end;
decode-side count guards are omitted when the limit exceeds int.MaxValue since
an int-typed count can never violate them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(discv5): make the consensus-only ENR drop policy injectable

The shared discv5 stack hard-dropped consensus-only node records in the
response handler, adapter acceptance check, and node source — fatal for a
consumer that wants CL peers. Discv5RecordFilter makes the policy injectable
with the execution-layer behavior as the default (unchanged); a beacon-chain
scope can register AcceptAll.

Needed by the embedded beacon chain driver plugin (#11976).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(discv5): dedicated IDiscv5RecordFilter, required everywhere

Replace the bool-parameterized filter class with an interface and two
implementations, and make the dependency non-optional at every record-handling
component (no null-coalesced defaults) so each discv5 instance states its
policy explicitly. The interface documents why the policy exists: discv5 is
one DHT shared by EL and CL nodes, and which records are useful is per
instance — the EL must drop consensus-only ENRs it can never dial over RLPx,
while a CL instance exists precisely to find them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ssz): lift the bitlist limit guard to ulong; drop a dead generator arm

Review follow-up: ValidateSszBitlistLimit now takes ulong like the list
variant, the BitList validation emission carries the UL suffix, and the
bitlist Encode emission clamps to int.MaxValue (the runtime parameter is
int-typed and unused). The Kind.List-with-BitArray arm could never fire —
such properties resolve to Kind.BitList — and is removed. Regression: the
huge-limit test now also covers a 2^40-limit bitlist.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(ssz): reject huge bitlist limits; address review feedback

- Generator's SszListAttribute mirror now takes `ulong limit`, matching the
  runtime attribute (flcl42).
- SszProperty parses the limit as `ulong` directly (the attribute is `ulong`,
  so the value is never `int`) and now rejects a `BitArray` list whose limit
  exceeds `int.MaxValue` with an SSZ003 generator error instead of silently
  clamping the int-typed Encode parameter — a BitArray cannot hold more than
  int.MaxValue bits (flcl42).
- Drop the now-unsupported huge-limit bitlist test type; add a generator
  diagnostic regression test for the rejection.
- Trim the IDiscv5RecordFilter <remarks> block (flcl42).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

@LukaszRozmej LukaszRozmej left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like LruCache changes

Comment thread src/Nethermind/Nethermind.Core/Caching/LruCache.cs Outdated
Comment thread src/Nethermind/Nethermind.Kademlia/DoubleEndedLru.cs
Comment thread src/Nethermind/Nethermind.Kademlia/LookupKNearestNeighbour.cs
Comment thread src/Nethermind/Nethermind.Kademlia/LookupKNearestNeighbour.cs
Comment thread src/Nethermind/Nethermind.Network/IPAddressClassifier.cs Outdated
Comment thread src/Nethermind/Nethermind.Network/NetworkNodeDecoder.cs
Comment thread src/Nethermind/Nethermind.Network/NetworkNodeDecoder.cs Outdated
Comment thread src/Nethermind/Nethermind.Runner.Test/Module/MainProcessingContextTests.cs Outdated
Comment thread src/Nethermind/Nethermind.Runner/configs/mainnet_archive.json
@flcl42 flcl42 marked this pull request as draft June 22, 2026 08:41
@flcl42 flcl42 marked this pull request as ready for review June 22, 2026 16:46
# Conflicts:
#	src/Nethermind/Nethermind.Init/Steps/InitializeNetwork.cs
#	src/Nethermind/Nethermind.Network.Discovery/DiscoveryApp.cs
#	src/Nethermind/Nethermind.Network.Discovery/DiscoveryConnectionsPool.cs
#	src/Nethermind/Nethermind.Network.Discovery/Discv4/Kademlia/KademliaAdapter.cs
#	src/Nethermind/Nethermind.Network.Discovery/Discv5/DiscoveryV5App.cs
@flcl42 flcl42 requested a review from LukaszRozmej June 22, 2026 17:01
@flcl42

flcl42 commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

@claude reivew all against master

@claude

claude Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Claude finished @flcl42's task in 8m 39s —— View job


PR Review: In-house discv5 — Full Review vs Master

  • Gather context and load rules
  • Diff against origin/master and identify changed areas
  • Review Kademlia core (generic routing, health tracking, bucket tree)
  • Review discv5 packet codec and session handling
  • Review discv5 KademliaAdapter
  • Review ENR handling, discv4 refactor, DI/module wiring
  • Review tests
  • Post final summary

Summary

This is a full review of all PR changes against master. All findings from previous rounds (1–4) have been addressed. Three new minor findings are noted below.


Previously Reported Findings — Status

All previously-reported High/Medium/Low findings from rounds 1–4 are resolved. Key items verified:

  • PruneLastBucketRefreshTicks properly prunes stale prefixes (was using unsound <= comparison)
  • PacketWorkerCount=4 parallel workers replace sequential processing in discv5 KademliaAdapter
  • ✅ Single malformed ENR in a NODES message no longer drops the whole message — TryDecodeNodeRecord handles per-record failures gracefully
  • _decodeMaskingAesPool is an ObjectPool<Aes> (was missing, now thread-safe across decode workers)
  • DefaultInboundMessageWorkerCount reduced from 16 to 4 in NettyDiscoveryHandler
  • ✅ Unsolicited discv4 responses (Pong/Neighbors/EnrResponse) no longer create session stats — IsResponse check gates the handler path
  • DisposingLruCache correctly auto-disposes evicted Session objects (clean inheritance over callback)
  • DiscoveryKademliaConfigFactory now passes real Node with actual endpoint (was placeholder 127.0.0.1:9999)
  • IteratorNodeLookup replaced by RandomWalkKademliaDiscovery — streaming results via Channel<TNode>, respects MinimumIterationDuration
  • SecP256k1Ecdh static delegate pinned to prevent GC collection of the EcdhHashFunction callback
  • ✅ ENR consensus-only rejection and unknown entry preservation work correctly

New Findings

Medium — PacketCodec.EncodeHandshake: sync-over-async with undocumented precondition

File: Nethermind.Network.Discovery/Discv5/Packets/PacketCodec.cs, line 89

NodeRecord currentNodeRecord = _nodeRecordProvider.GetCurrentAsync().GetAwaiter().GetResult();

This is a blocking .GetAwaiter().GetResult() on a ValueTask-returning async method. It is safe in practice because NodeRecordProvider pre-warms its cache during app initialization (ipResolver.Resolve() completes before discv5 starts), so by the time this code executes the Task<NodeRecord> is already completed and GetResult() returns immediately without blocking a thread.

However, the precondition is undocumented. A future refactor that changes initialization order or lazy-starts the provider would silently introduce a thread-pool thread block. Options:

  • Preferred: document the invariant in a <remarks> comment — "safe because NodeRecordProvider is pre-warmed before discv5 handshakes can occur."
  • Alternative: make EncodeHandshake async and propagate await up the call chain.

Low — LruCache.GetValues() uses [Synchronized] inconsistent with McsLock

File: Nethermind.Core/Caching/LruCache.cs

GetValues() is annotated [MethodImpl(MethodImplOptions.Synchronized)] (Monitor on this), while all other LruCache methods use McsLock _lock. This is a pre-existing inconsistency, now more visible since the PR modifies LruCache (adding the Evict virtual and NotifyEvictedValues method). It won't cause correctness issues in typical usage since GetValues() is a snapshot method, but mixing two lock types on the same state is fragile. Out of scope to fix here, but worth a follow-up.


Low — RegisterKnownRecord re-parses ENR string on every outbound Ping/FindNeighbours

File: Nethermind.Network.Discovery/Discv5/Kademlia/KademliaAdapter.cs, RegisterKnownRecord

private void RegisterKnownRecord(Node node)
{
    if (string.IsNullOrEmpty(node.Enr)) return;
    NodeRecord record = NodeRecord.FromString(node.Enr, _ecdsa);
    TrySetKnownRecord(record);
}

TrySetKnownRecord uses a double-check pattern to skip records with lower sequence numbers than already known. For a node already tracked in _knownRecords, RegisterKnownRecord still re-parses the ENR string (RLP decode + ECDSA signature verify) on every outbound call before discovering the record is already known. A _knownRecords.TryGetValue guard before parsing would avoid the redundant cryptographic work for already-known nodes. This is a minor hot path concern rather than a correctness issue.


Recent Commits — Verified

  • 0640843d — Discv4 hardening: unsolicited responses gated by IsResponse + HandleViaMessageHandlers. Test OnIncomingMsg_unsolicited_response_should_not_create_session_stats covers the fix. ✅
  • dc729689 — LRU eviction via inheritance (DisposingLruCache<TKey, TValue> : LruCache). Cleaner than callback; Clear() correctly calls NotifyEvictedValues outside the lock. ✅
  • ca09665eNodesResponseHandler concurrent batch handling + Distances using InlineArray(3) with pool fallback for larger counts. ShouldCollectConcurrentBatchesOnce test validates the concurrent case. ✅
  • cd50eaf5IteratorNodeLookup deleted, RandomWalkKademliaDiscovery added. Streaming via Channel<TNode>, MinimumIterationDuration=1s between iterations, LookupNodes in LookupKNearestNeighbour for the streaming path. ✅
  • dddbe6b2DiscoveryKademliaModuleBase and DiscoveryKademliaConfigFactory fixed to pass real Node with actual endpoint. ✅

Overall Verdict

The PR is in excellent shape. The architecture is sound, all previously-reported issues are resolved, and the recent commits are well-targeted improvements. The three findings above are minor — the Medium item is a documentation gap (not a runtime bug), and both Low items are efficiency/consistency observations. The PR is ready to merge.

flcl42 added 5 commits June 22, 2026 20:06
# Conflicts:
#	src/Nethermind/Nethermind.Network.Discovery.Test/E2EDiscoveryTests.cs
#	src/Nethermind/Nethermind.Network.Discovery/DiscoveryApp.cs
#	src/Nethermind/Nethermind.Network.Discovery/Discv4/DiscV4KademliaModule.cs
@flcl42 flcl42 merged commit 9a79c71 into master Jun 23, 2026
558 checks passed
@flcl42 flcl42 deleted the generic-kad-2 branch June 23, 2026 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants